Manufacturing intelligence service system connected to mes in smart factory

ABSTRACT

A manufacturing intelligence service system connected to an MES in smart factory is provided. The smart factory manufacturing intelligence service system connected to an MES includes a Manufacturing Execution System (MES) having a machine vision of a production line of each manufacturing company to provide the product ID and a product information and a defect information including scratch or defect of a product; a cloud server connected to the at least one Manufacturing Execution System (MES); and an agent server connected to the cloud server, and the cloud server provides the product ID and the product information and product defect information of a connected machine vision production line of a manufacturing company product to the user terminal through the agent server.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to Korean Patent Application No.10-2021-0157125, filed on Nov. 15, 2021, the disclosures of which isherein incorporated by reference in its entirety.

This invention was supported by a grant of the National IT IndustryPromotion Agency (NIPA, No. A1308-22-1008).

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a smart factory manufacturingintelligence service system, and more particularly, to a manufacturingintelligence service system connected to a Manufacturing ExecutionSystem (MES) in smart factory, which provides product ID, productinformation and defect information of products of each manufacturingcompany accumulated and stored in a cloud server, from an agent serverto a user terminal, through the cloud server connected to at least oneManufacturing Execution System (MES) having a machine vision inspectionsystem for detecting defects of products in a product surface defectinspection in the factory automation (FA) process of an industrialcompany.

Background of the Invention

Recently, A machine vision (MV) inspection system is technologies ofcombining robots, machines with software technologies of vision systems,and it is used in product surface defect inspection, PCB defectinspection, or LED chip package inspection to detect defects of productsin various fields such as wafer, display, and PCB inspection, LED chipsemiconductor package inspection, and the like in the factory automation(FA) process of an industrial company. The machine vision inspectionsystem is used to detect defects of products by inspecting defects onthe surface of mass-produced products.

The machine vision inspection system includes an optical light (LED orhalogen light), a high-resolution camera, camera interface equipment[Frame Grabber, Gigabit Ethernet (GigE), IEEE 1394, camera link, USB3.0], a vision image processing board being in charge of I/O and motioncontrol of an industrial PC, and vision image processing software.

A PC-based machine vision inspection system is a vision inspectionsystem including an optical light (LED or halogen light), a camera, acamera interface [Frame Grabber, Gigabit Ethernet (GigE), IEEE 1394,camera link, or USB 3.0], and a vision image processing board mounted ona computer, and detects foreign materials, scratches, and pattern errorson the surface of a product by image processing of vision inspectionimage processing software for inspection on a product surface image.

The vision inspection image processing software provides the functionsof frame grabbing, image processing, feature extraction, defectdetection, and control, and the vision inspection system of a computerconnected to a camera detects defects of a product by inspecting defectson the surface of the product in real time, comparing feature values ofa normal image and an inspection image by using software having an imageprocessing and defect detection algorithm, and measuring foreignsubstances, scratches, pattern errors, dented defects, and defectlocations.

The camera interface that connects the camera and the PC uses FrameGrabber, Gigabit Ethernet (GigE), IEEE 1394a, IEEE 1394b, camera link,or USB 3.0. Table 1 shows the machine vision camera interface.

The vision inspection system performs image processing by using anexisting frame grabber, or using a camera interface such as GigabitEthernet (GigE), CoaXPres, or the like to perform high-resolutionprecise measurement of μm level and to process large-capacity data at ahigh speed.

TABLE 1 Gigabit Ethernet Category (GigE) IEEE 1394a IEEE 1394b Cameralink Cable length 100 m 4.5 m (up to 72 m 4.5 m (up to 72 m Maximum 15 min case of using in case of using a repeater) a repeater) Bandwidth 100Mb/s 32 Mbytes/s 64 Mbytes/s 255 Mbytes/s(base configuration) 680Mbytes/s(full configuration) Bit rate 1000 Mb/s 400 Mb/s 800 Mb/s >2000Mb/s Standard GigE Vision IEEE 1394 Trade IEEE 1394 Trade AIA cameralink Standard Association DCAM Association DCAM standard StandardStandard Interface Gigabit IEEE 1394a Special frame board Ethernet boardgrabber board Maximum Unlimited 16 cameras 16 cameras 2 cameras pernumber of (DCAM) (DCAM) frame grabber cameras Cable IndustrialIndustrial and Industrial and Industrial and consumer consumer consumer

FIG. 1 is a view showing the configuration of a vision inspection systemhaving a sensor, an ID reader, a camera, and an optical light in aconveyor belt production line provided with an encoder.

As prior art 1 related thereto, “Multiple optical module visioninspection system” is registered in Korean Patent Registration No.10-1772673.

The multiple optical module vision inspection system includes amechanism unit; a camera unit provided with a computer having akeyboard, a mouse, and a monitor on the front side of the mechanismunit, and provided with at least one high-resolution multi-cameraconstituting a single camera, a line scan camera, or an area scan cameraon a fixed support; an optical light unit installed under the cameraunit; an inspection stage located under the camera unit and the opticallight unit to place an inspection target on an XY-stage; a base placedunder the inspection stage; an anti-vibration facility of a vibrationreduction air cylinder structure placed under the base; a frame forsupporting the left and right sides of vision inspection equipment; anda stage unit transfer module for controlling movement of XYZ position.

The camera unit includes at least one multi-camera constituting a singlecamera, a line scan camera, or an area scan camera on a fixed support,in which each camera is connected to a PC through a camera interface,each PC is connected to a main server computer via LAN and TCP/IPthrough a network hub, the main server computer is connected to anencoder/distributor via LAN and TCP/IP through a network hub, and theencoder/distributor is connected to the multi-cameras consisting of aline scan camera or an area scan camera.

In the camera unit, each camera uses a precision measurement camera witha high resolution of 10 to 100 μm (pixel size), which is improved asmuch as 100 times than that of a general CCD camera, for precisemeasurement of product surface defects.

1) ID reader (reads a DPM code or a 1D code or 2D code attached to aproduct): The ID reader reads the ID of an inspection target transferredto a conveyor belt of a production line in a factory automation process,or the ID of a product placed on an inspection stage which is an XYstage by a loader of an inspection target transfer robot, and transmitsthe detected product ID to the computer.

2) Optical light: In the vision inspection system, a white or red LEDlight or a halogen light having an optical fiber guide is used as theoptical light, and LEDs are used in the embodiment of the presentinvention.

In the case of using LED light, white LEDs arranged in a row and a lightcontroller, or a ring LED having a plurality of LEDs surrounding acamera lens and a light controller is used.

3) Camera

A vision camera, a line scan camera, or an area scan camera is providedwith multiple cameras, and each camera uses a TDI camera capable ofmaking a measurement as precise as a high resolution of 10 to 20 μmunits, which is improved as much as 100 times than that of a general CCDcamera, for precise measurement of product surface defects. In addition,when measurement of a camera does not require a precision as much as 10to 20 μm units, a CCD camera having a resolution lower than that of aTDI camera or a CMOS image sensor (CIS) is used.

4) Encoder

An optical light and a line scan camera or an area scan camera equippedwith a high-resolution multi-camera are located while being fixed to asupport on a conveyor belt system operated by driving of a step motordriver and a step motor (servo motor) connected to a computer in thefactory automation process of an industrial company. The encodermeasures an exact amount of transfer of the servo motor when a conveyorbelt operates in a production line of a factory.

The mechanism unit further includes an inspection target transfer robotfor placing an inspection target on the inspection stage (XY stage) by aloader.

The mechanism unit may further include an ID reader for reading a DPMcode, a barcode, or a QR code attached to a product as an ID of aninspection target transferred to the conveyor belt of a production linein a factory automation process, or an ID of a product placed on theinspection stage (XY stage) by the loader of an inspection targettransfer robot, and transmitting the detected product ID to thecomputer.

The stage unit transfer module controls the line scan camera to move inthe X-axis direction by a linear motor, move the position of theinspection stage in the Y-axis direction by a linear motor, and move thecamera unit in the vertical direction (Z-axis direction) along Z1, Z2,and Z3 axes.

The linear motor is operated by a motor driving unit connected to acontrol computer, and uses PID control to control the speed of the motorand movement position.

The anti-vibration facility uses an anti-vibration system that uses anair cylinder structure on the bottom of a grid beam under the base toreduce shaking and vibration during precise measurement of a visioninspection system using a line scan or area scan camera for precisemeasurement of 10 to 100 μm pixel size, i.e., installs a granite plateand an isolator to block external vibration and prevent internalvibration.

The machine vision filters defects of a product by analyzing defectssuch as foreign substances, scratches, pattern errors, or the like onthe display of a product surface, and locations of the defects by imageprocessing [(1) image acquisition, (2) image binarization, (3) imageprocessing, (4) image analysis, (5) image interpretation] of a visioninspection image processing algorithm of a computer connected throughthe mechanism unit and the camera interface (Frame Grabber, GigabitEthernet (GigE), IEEE 1394a, IEEE 1394b, camera link, or USB 3.0)connected to a line scan camera or an area scan camera.

For example, in the case of inspecting defects on the surface of aTFT-LCD panel, a TFT-LCD image acquired by an industrial PC from animage acquired using a line scan camera through the camera interfacerequires fast image processing time in the production process. As theTFT-LCD images captured by the camera have a repeating pattern, themachine vision inspection system may easily detect defective areas of aproduct such as scratches, surface defects, and the like by using thedifference in the image from those of adjacent patterns.

As prior art 2 related thereto, Korean Patent Publication No.10-2019-0063839 provides “Method and system for machine vision-basedquality inspection using deep learning in manufacturing process”.

FIG. 2 is a view showing the operation flow of generating a conventionalclassifier based on deep learning.

The machine vision-based quality inspection method using deep learningin a manufacturing process includes the steps of generating a productimage for learning; training a classifier that classifies good and badproducts with the generated product image for learning; and determiningwhether a product is good or defective product by using the trainedclassifier. Accordingly, feature values of data of a determinationtarget may be found by learning itself, and a machine vision-basedinspection may be performed even on an inspection area that relies onmanual inspection because it is difficult to formulate defects.

In order to determine whether a product is good or defective product byquality inspection, first, an inspection target product is photographedusing a camera after aligning the position of the inspection camera, anda region of interest (ROI) is extracted from an image photographed bythe camera and stored a ROI image in a memory. The target images in theregion of interest are cropped into overlapped small images of apredetermined size, like image preprocessing of generating a learningmodel.

A quality inspection device performs inspection on each fragmented imageby using a classifier. A deep neural network algorithm outputs aclassification number and a probability value of a nearest image, amongthe learning data used when the classifier is generated by learning, asa classification result. It is confirmed through a deep learning-basedclassifier whether each fragmented image is good or defective image withreference to the classification table with the highest probability at acorresponding index, and a result of the determination is displayed. Atthis point, in the case of a defective product, the product isdetermined as defective, and the defective area is marked.

As prior art 3 related thereto, “Inspection apparatus and method formachine vision system” is registered in Korean Patent Registration No.10-11827680000.

The inspection apparatus for a machine vision system includes asupporter including one or more lights that illuminate to an inner partthereof and having a predetermined shape; a portable terminal having acamera and photographing an inspection object positioned in thesupporter while being fixed to the supporter; and a control unit settinga photographing condition of the portable terminal in accordance with aninstruction of the portable terminal and controlling the lights inaccordance with the set photographing condition.

As prior art 4 related thereto, “Machine vision based electroniccomponent inspection system” is registered in Korean Patent RegistrationNo. 10-1688641.

The machine vision based electronic component inspection system isconfigured to include a housing equipped with a selection plate insidethereof so that various kinds of electronic components supplied inplurality are seated unaligned and automatically selected for qualityinspection; a location moving device installed to move as much as a setdistance on the top of the selection plate, a camera device installed inthe location moving device to photograph the electronic componentsrandomly arranged on the selection plate; an analysis device installedin the housing to calculate the location and angle of each electroniccomponent on the basis of information on images photographed by thecamera device; a gripper device installed in the location moving deviceto adsorb the electronic components randomly arranged on the selectionplate and sequentially move the electronic components to a designatedposition on the selection plate based on the calculated values such asthe location and angle of the electronic components determined by theanalysis device; an inspection device installed in the location movingdevice to perform quality inspection on the electronic components moved,aligned, and fixed by the gripper device in accordance with the capacityof each target; and a control device for controlling operation of thelocation moving device, the camera device, the analysis device, thegripper device, and the inspection device,

wherein the gripper device is provided with a rotation device forrotating the electronic components while being individually adsorbed,and the rotation device rotates each of the electronic componentsadsorbed to the gripper device based on the calculated values of theanalysis device, so that each of the electronic components randomlyarranged on the selection plate is positioned in a right direction onthe inspection device to perform the quality inspection.

As prior art 5 related thereto, “Machine vision inspection device havinga machine vision modular software using artificial intelligence, adriving method thereof, and a computer-readable recording mediumthereof” is registered in Korean Patent Registration No. 10-21089560000.

The prior art 5 relates to a machine vision inspecting device having amachine vision modular software using artificial intelligence, a drivingmethod thereof, and a computer-readable recording medium thereof, andthe machine vision inspecting device having a machine vision modularsoftware using artificial intelligence according to an embodiment of thepresent invention includes: a storage unit for storing learning sampledata of a first volume of a plurality of inspection items related to adesignated product in a different way for each user for the sake ofartificial intelligence-based vision inspection, and a control unit forperforming vision inspection on the designated product by performingdeep learning based on artificial intelligence for each inspection itemon the basis of differently stored learning sample data of the firstvolume and a captured image provided by a photographing device of aproduction line, and expanding the learning sample data from the firstvolume to a second volume on the basis of a result of the visioninspection.

As prior art 6 related thereto, “Machine vision-based quality inspectionmethod and system utilizing deep learning in manufacturing process” ismade public in Korean Patent Publication No. 10-2019-0063839.

The machine vision-based quality inspection method using deep learningin a manufacturing process generates a learning product image, enables aclassifier to be learned for classifying a good product and a defectiveproduct through the generated learning product image, and determineswhether a product is a good product or a defective product by using thelearned classifier. Therefore, it is possible to find the feature valueof data of a product to be classified on the basis of learning ofoneself, such that it is possible to carry out the machine vision-basedinspection even on an inspection area that relies on a manual inspectiondue to the difficult of formalizing defects.

As prior art 7 related thereto, “Control variable setting device in asemiconductor vision inspection system based on deep learning and amethod thereof” is made public in Korean Patent Publication No.10-2019-0067439.

The operation method of the control variable setting device in asemiconductor vision inspection system based on deep learning comprisesthe steps of: receiving information about a plurality of controlvariables controlling operation of an image acquisition device; usingthe number of control variables included in the received informationabout the control variables and the number of variable values of thecontrol variables and forming a control variable set; receiving imagesacquired by the image acquisition device that is respectively reflectingthe variable values a plurality of the control variables correspondingto each element of the formed control variable set with respect to atleast one vision inspection sample; and using a previously trained deeplearning-based neural network to determine a final image among thereceived images, and acquiring respective final variable values aplurality of the control variables for acquiring the determined finalimage,

wherein each element of the formed control variable set includes all ofthe control variables and two arbitrary elements different from eachother of the formed control variable set are corresponded to havedifferent variable values among at least one identical control variable.

In the initial stage in which a smart factory is not constructed at all,it is not easy for a company in the manufacturing industry field tocomputerize information on the inventory processing process completelymanually on the spot by providing work management to store raw materialsin a warehouse, management to obtain and place an order, productionplan, work management by production order, LOT management, processmanagement, quality management, warehousing/releasing/inventorymanagement, and sales performance management.

A smart factory is constructed in connection with a ManufacturingExecution System (MES), a vision inspection system, and a QualityManagement System (QMS). The MES provides real-time camera visioninspection monitoring, control, logistics and work history trackingmanagement, and product defect management in a manufacturing process.The product vision inspection system of products that have a barcode, aQR code, or a 13.56 MHz RFID tag attached to product filters defectiveproducts manufactured and produced along the conveyor belt in real timein accordance with a manufacturing process of the production line bydetecting defects in products generated as atypical data by the machinevision system.

However, in the vision inspection system, a computer vision recognitionerror occurs due to a defect in an atypical pattern of a product,diffuse reflection of a metal surface, and vibration through cameraimages in real time.

It costs too much for a small and medium-sized company to actuallyconstruct a smart factory having an existing machine vision defectdetection system and install conveyor belts, encoders, cameras, andvision systems in a manufacturing process and product production lines.At the standpoint of a company that has already adopted a machine vision(MV), installation of a deep learning vision system may be a duplicateinvestment. In addition, it is difficult for small and medium-sizedcompanies, other than large companies equipped with manufacturingprocesses, production lines, and machine learning vision systems thatconstruct a smart factory, to construct conveyor belts and deep learningvision systems that cost more than 100 million Korean Won in aproduction line in practice, and to adopt an expensive deep learningvision inspection system using artificial intelligence programming inreality.

Prior Art-Patent Documents

-   (Patent Document 1) Korean Patent Registration No. 10-1772673    (Registration date: Aug. 23, 2017), “Multiple optics vision    inspection system”, APS Co., Ltd.-   (Patent Document 2) Korean Patent Publication No. 10-2019-0063839    (Publication date: Jun. 10, 2019), “Method and system for machine    vision-based quality inspection using deep learning in manufacturing    process”, Korea Electronic Components Research Institute-   (Patent Document 3) Korean Patent Registration No. 10-11827680000    (Registration date: Sep. 7, 2012), “Examining apparatus and method    for machine vision system”, Mvision Co., Ltd.-   (Patent Document 4) Korean Patent Registration No. 10-1688641    (Registration date: Dec. 15, 2016), “Machine vision based electronic    components inspection systems”, Dongseo University Industry-Academic    Cooperation Foundation, KMsys Co., Ltd.-   (Patent Document 5) Korean Patent Registration No. 10-21089560000    (Registration date: May 4, 2020), “Apparatus for performing    inspection of machine vision and driving method thereof, and    Computer Readable Recording Medium”, Mann Company-   (Patent Document 6) Korean Patent Publication No. 10-2019-0063839    (Publication date: Jun. 10, 2019), “Machine vision-based quality    inspection method and system using deep learning in manufacturing    process”, Korea Electronics Technology Institute-   (Patent Document 7) Korean Patent Publication No. 10-2019-0067439    (Publication date: Jun. 17, 2019), “Control value setting apparatus    and method of semiconductor vision inspection system based on deep    learning”, Electronics and Telecommunications Research Institute.

SUMMARY OF THE INVENTION

To solve the above-described problems in the related art, and an objectof the present invention is to provide a manufacturing intelligenceservice system connected to an MES in smart factory, which is providedwith a cloud server connected to at least one Manufacturing ExecutionSystem (MES) having a machine vision inspection system for detectingdefects of products in various fields such as product surface defectinspection or the like in the factory automation (FA) process of anindustrial company, and provides a product ID and a product informationand defect information of products of a plurality of manufacturingcompany products accumulated and stored in the cloud server from anagent server to a user terminal through the cloud server.

To accomplish the above object, according to one aspect of the presentinvention, there is provided a manufacturing intelligence service systemconnected to a Manufacturing Execution System (MES) in smart factory,the system comprising: at least one Manufacturing Execution System (MES)having a machine vision of a production line of each manufacturingcompany, recognizing a product ID, for providing the product ID, aproduct information and a defect information including scratch or defectof a product through middleware; a cloud server connected to the atleast one Manufacturing Execution System (MES); and an agent serverconnected to the cloud server, wherein the agent server is connected tothe cloud server and provides the product ID, a product information andproduct defect information of a machine vision production line of amanufacturing company product to the user terminal through the agentserver from the cloud server.

A manufacturing intelligence service system connected to an MES in smartfactory, which includes a cloud server connected to at least oneManufacturing Execution System (MES) having a machine vision inspectionsystem for detecting defects of products or product surface defectinspection in various fields such as wafer, display, and PCB defectinspection, LED chip semiconductor package inspection. The cloud serverhas an effect on providing user with product ID, product information andproduct defect information of a manufacturing company productaccumulated and stored in the cloud server, from the cloud server to theuser terminals through the agent server, thereby effectively providingmany companies with smart factory manufacturing intelligence servicedata.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view showing the configuration of a vision inspection systemhaving a sensor, an ID reader, a camera, and an optical light in aconveyor belt production line provided with an encoder.

FIGS. 2A, 2B and 3 are views showing main functions of a manual visioninspection machine having a camera, a light, and a controller connectedto a PC to determine defects of products by detecting defects (foreignsubstances, or scratches, etc.) of the products.

FIG. 4 is a view showing a machine vision manufacturing intelligenceplatform using product defect image remote learning by using a deeplearning algorithm.

FIG. 5 is a view showing an AI view of vision inspection (stains, orscratches), product shape inspection (unpunched, or size defect), andblob inspection (determine whether or not plated) of a PC by using asensor and a camera in a deep learning-based machine vision platform ina vision inspection method using product defect image remote learning.

FIG. 6 is a view showing the configuration of a cloud server of a smartfactory.

FIG. 7 is a view showing the configuration of a manufacturingintelligence service system connected to an MES in smart factoryaccording to the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, example embodiments of the present invention will bedescribed in detail with reference to the accompanying drawings. In thedescription of the present invention, when it is determined that adetailed description of a related known technology or a knownconfiguration may unnecessarily obscure the subject matter of thepresent invention, the detailed description will be omitted. Inaddition, when a reference numeral of a drawing indicates the sameconfiguration, the same reference numeral is assigned in differentdrawings.

A manufacturing intelligence service system connected to an MES in smartfactory constructs a smart factory in the factory automation (FA)process of an industrial company, together with one or moremanufacturing companies, and through a cloud server connected to atleast one Manufacturing Execution System (MES) having a machine visioninspection system that inspects product surface defects and detectsdefects of products in various fields, the smart factory manufacturingintelligence service system provides user with product ID, productinformation and defect information of products of each manufacturingcompany accumulated and stored in the cloud server from an agent serverto a user terminal through the agent server.

-   -   Recognition of a barcode, a QR code, and a 13.6 MHz code        attached to a product (recognition of product ID)    -   Alignment of parts in an assembly process, stacking alignment,        and product surface defect inspection (machine vision)    -   Provide an AI View of vision inspection (stains, scratches),        product shape inspection (size defect), and blob inspection        (determine whether or not plated) of a PC using sensors and        cameras on a deep learning-based machine vision platform

FIGS. 2A and 2B are views showing main functions of a vision inspectionmachine having a camera, a light, and a controller connected to a PC todetermine defects of products by detecting defects (foreign substances,or scratches, etc.) of the products.

* Smart Factory Manufacturing Intelligence Service System Connected toMES

1) A deep learning vision inspection function is modularized andinspects a product (normal/defective) in association with an existingmachine vision (MV) inspection system. The deep learning algorithm ofmachine vision image analysis SW uses an CNN algorithm, and uses any oneof AlexNet, ZFNet, VGGNet, GoogLeNet, and ResNet.

A deep learning tool uses any one of Tensorflow, Keras, Caffe, andPyTorch.

2) Construct data for deep learning using a test image acquired by themachine vision (MV) inspection system and a determination result forreference.

* Main Functions

1. Machine vision defect detection system: When there is existingmachine vision equipment, only a deep learning vision defect detectionmodule may be adopted.

2. Machine vision interface: Receives a test sample image and adetermination result for reference from existing machine visionequipment.

3. Deep learning: Deep learning is carried out with the received imageand the determination result for reference

4. Deep learning determination: Determination is carried out with alearning model generated in 2 for a sample image for detecting defects.

5. Retraining of deep learning: A field worker makes a finaldetermination seeing the determination result of 3 and, and this resultis reused as a deep learning material.

6. When the steps are sufficiently performed, accuracy of determiningthe deep learning vision inspection is enhanced, and the user does notrequire to separately make a determination.

FIG. 3 is a view showing wiring of a vision system connected to acamera, a light, a control box, a PC, or a PLC.

An AI machine vision inspection system through product defect imageremote learning includes a PC 100, a PLC 110, a camera 120, a sensor130, a light 140, and a controller 170.

The controller 170 is connected to the camera 120, the sensor 130, andthe light 140, and the PC 100 connected to the camera 120 may beconnected to the PLC 110.

There are provided a camera 120, a light 140, and a controller 170connected to the PC 100, and a manual vision inspection machine thatdetects defects of products (foreign materials, scratches, etc.) anddetermines defective products (defects, scratches, foreign materials,etc.) further includes a sensor 130 for additionally providing a triggerinput to the camera 120.

In the AI machine vision inspection system, the lighting 140 may be useda white or red LED light, or a halogen light having an optical fiberguide is used as the light 140, and in an embodiment, an LED light and alight controller are used.

In case of using an LED light, white LEDs arranged in a row and a lightcontroller, or a ring LED including a plurality of LEDs surrounding acamera lens and a light controller is used. The lighting 140 may be usedring LED illumination, top/left-top/top/right-top tilt angleillumination, backlight illumination, or the like.

Additionally, the PC 100 having a deep learning-based vision imageprocessing SW further includes a PLC 110 connected through an Ethernetcable.

The machine vision image analysis SW for the camera image data of theproduct uses i) grayscale image processing, or ii) a deep learningalgorithm, and a grayscale image, an RGB image, an HSI image, a YCbCrimage, a JPEG image, a TIFF image, and a GIF image may be applied as thecamera image. The deep learning algorithm detects objects having defectsof foreign substances and scratches in the camera image data anddetermines whether the product is defective using any one ofConvolutional Neural Network (CNN), Recurrent Convolutional NeuralNetwork (R-CNN), Fast RCNN, Faster RCNN(Region based ConvolutionalNeural Network), You Only Look Once (YOLO), and Single Shot Detector(SSD) algorithms.

The deep learning algorithm of the machine vision image analysis SWdetects objects in an image and determines whether the product isdefective by using any one of Convolutional Neural Network (CNN),Recurrent Convolutional Neural Network (R-CNN), Fast RCNN, FasterRCNN(Region based Convolutional Neural Network), YOLO(You Only LookOnce), and SSD (Single Shot Detector) algorithms.

The deep learning algorithm of the machine vision image analysis SW ofthe edge platform uses a CNN algorithm, and uses any one among AlexNet,ZFNet, VGGNet, GoogLeNet, and ResNet. The deep learning algorithm usesthe CNN algorithm to extract and classify features of an image, extractdefective objects (foreign substances, defects, or scratches) bycomparing the features with the learning data of accumulated defectimages of the learning model, and transmit an image containing thedefective objects to the service platform, and the service platformdetermines whether the product is defective.

A barcode, a QR code, or a 13.56 MHz RFID tag is attached to theproduct, and a barcode reader, a QR code scanner of an industrial PC, ora 13.56 MHz RFID reader is respectively used.

Additionally, the PC may further include a barcode reader and arecognition module for recognizing a barcode attached to a product whenthe barcode is attached to the product.

Additionally, the PC may further include a QR code recognition modulefor recognizing a QR code attached to a product when the QR code isattached to the product.

Additionally, the PC may further include a SW module connected to a13.56 MHz RFID reader through “product code transmission middleware”when a 13.56 MHz RFID tag is attached to a product.

The middleware includes product code transmission middleware fortransmitting information on any one of a barcode, a QR code, and a 13.56MHz RFID tag corresponding to the extracted model information attachedto the product recognized by the barcode reader, the QR code recognizer,or the 13.56 MHz RFID reader to the cloud server; and deep learningmiddleware provided with an atypical defect determination learning modelfor receiving atypical defect process data transmitted from the machinevision system and detecting atypical defective images by comparing theatypical defect process data with the defective image learning data(foreign materials, scratches) accumulated and stored by a deep learningmodel training system, and transmitting data on the result of foreignmaterial existence inspection, shape inspection, and normal/defectivedetermination performed on the camera image data by a deep learningshape determination system and an AI deep learning module to cloudserver.

Additionally, the PC may be connected to a control robot through a robotinterface to control movement of the product after performing 2D visioninspection based on deep learning.

FIG. 4 is a view showing a machine vision manufacturing intelligenceplatform using product defect image remote learning by using a deeplearning algorithm.

The machine vision manufacturing intelligence platform supports a deeplearning machine vision platform that can quickly respond to a newproduct and occurrence of an exceptional defect type by making existingmachine vision inspection equipment intelligent in a hybrid form. Acompany that adopts the smart factory solution may secure qualityinspection intelligence of products produced in real time.

1. The machine vision manufacturing intelligence platform provides amanufacturing intelligence edge platform to a company by mounting an AIsolution that supports intelligence of machine vision inspectionequipment that has already been adopted.

2. When an exceptional defect type is generated in the machine visioninspection data (camera image data of a product) of a company that hasadopted a smart factory solution or a new product needs to be inspected,the service platform provides manufacturing intelligence to the edgeplatform by using the deep learning solution.

3. The service platform continuously provides manufacturing intelligenceservice to enhance intelligence of an existing edge platform by usingmachine vision manufacturing intelligence through learning ofmanufacturing common data to similar business types.

4. The machine vision system provides a product inspection platform inaccordance with manufacturing intelligence based on learning of defectdata in association with the MES system.

The machine vision manufacturing intelligence platform is provided witha service platform connected to the edge platform including a defectinspection module, a defect determination module, and a learning-purposemanufacturing data transmission module through middleware, in which thedefect inspection module reads a product ID and provides shapedetermination inspection/foreign substance inspection/scratch inspectionof camera image data, and the service platform provides defectdetermination manufacturing intelligence and defect predictionmanufacturing intelligence, and uses a deep learning algorithm based onthe learning data.

The product is attached with any one of a barcode, a QR code, or a 13.56MHz RFID tag.

Additionally, the PC further includes a barcode reader and a recognitionmodule for recognizing a barcode attached to a product when the barcodeis attached to the product.

Additionally, the PC further includes a QR code recognition module forrecognizing a QR code attached to a product when the QR code is attachedto the product.

Additionally, the PC further includes a SW module connected to a 13.56MHz RFID reader through “product code transmission middleware” when a13.56 MHz RFID tag is attached to the product.

A vision inspection system through product defect image remote learningincludes a machine vision inspection system connected to a camera, asensor, an LED light, and a controller and provided with machine visionimage analysis SW, and a reader (barcode reader, QR code recognizer, or13.56 MHz RFID reader) connected to the computer (PC) of the machinevision inspection system reads a product ID (barcode, QR code, 13.56 MHzRFID tag), and the vision inspection system includes an edge platform ofthe agent server provided clients with a defect inspection module thatprovides shape determination inspection/foreign substanceinspection/scratch inspection of camera image data of a product, adefect determination module, and a learning-purpose manufacturing datatransmission module; middleware connected to the edge platform tointerwork with the service platform of the cloud server; and a serviceplatform connected to the edge platform through the middleware toprovide defect determination manufacturing intelligence and defectprediction manufacturing intelligence, detect an atypical defectiveimage by comparing with accumulated and stored defective image learningdata (training data set of a defective image including foreignsubstances or scratches), and provide vision inspection through productdefect image remote learning using an AI deep learning algorithm thatprovides result data of foreign substances inspection, shape inspection,and normal/defect determination of camera image data of a product.

The edge platform of the agent server includes a defect inspectionmodule (shape determination inspection, foreign material inspection,scratch inspection, specification information collection based oninspection data, inspection prediction analysis screen, inspectionresult screen, good/defective inspection result determination labelingstorage and transmission), a defect determination module (determinationlabeling, threshold analysis), a defect prediction module (shapeprediction analysis, foreign substance prediction analysis, scratchprediction analysis, prediction rule correlation coefficient module),and a learning-purpose manufacturing data transmission module(manufacturing data storage and transmission module).

The service platform of the cloud connects to a Manufacturing ExecutionSystem (MES) and server shares defect determination manufacturingintelligence, defect prediction manufacturing intelligence, andmanufacturing data for learning with the edge platform, and the serviceplatform of the cloud server is provided with a manufacturingintelligence service module that provides development intelligence afterdeep learning, and includes a Scikit-learn Engine, a CNN, an RNN, anaudio encoder, a DB for storing the manufacturing data for learning, anda communication module on the Python framework.

A barcode, a QR code, or a 13.56 MHz RFID tag is attached to a product,and the middleware includes: product code transmission middleware fortransmitting information on any one of a barcode, a QR code, or a 13.56MHz RFID tag of a product recognized by a barcode reader, a QR coderecognizer, or a 13.56 MHz RFID reader from cloud server to the userterminal via the agent server; and deep learning middleware providedwith an atypical defect determination learning model for receivingatypical defect process data transmitted from a machine vision systemand detecting atypical defective images by comparing the atypical defectprocess data with the defective image learning data (foreign substances,or scratches) accumulated and stored by a deep learning model trainingsystem in accordance with a vision inspection method through productdefect image remote learning, and transmitting data on the result offoreign substance existence inspection, shape inspection, andnormal/defective determination performed on the camera image data by adeep learning shape determination system and an AI deep learning modulefrom the service platform of the cloud server to the agent server.

The system further includes an inspection stage located under the cameraunit and the optical light unit to place an inspection target on anXY-stage; a base on which the inspection stage is placed; ananti-vibration facility of a vibration reduction air cylinder structureplaced under the base; a frame for supporting vision inspectionequipment; and a stage unit transfer module for controlling movement ofXYZ position.

The camera is connected to a PC through a camera interface (framegrabber, Gigabit Ethernet (GigE), IEEE 1394, camera link, or USB3.0),and the PC is connected to a main server computer through LAN and TCP/IPvia a network hub.

FIG. 5 is a view showing an AI view of vision inspection (stains, dents,or scratches), product shape inspection (unpunched, deformation defect),and blob inspection (determine whether or not plated) of a PC by using asensor and a camera in a deep learning-based machine vision platform ina vision inspection method using product defect image remote learning.

The sensor generates a trigger input and transmits it to the camera, andthe camera generates a digital output by controlling the lightingstrobe, and an image sensor generates and transmits image data to thePC.

The PC performs image data inspection (stains, dents, or scratches),product shape inspection (unpunched, and deformation defect), and blobinspection (determine whether or not plated) as needed.

The inspection (stains, dents, or scratches) determines defects of animage in real time based on a classification threshold after registeringgood and defective images.

The product shape inspection (unpunched, and deformation defect)determines whether or not the shape of a product is changed from theshape and size of a product based on an image of a good product.

The blob inspection (determine whether or not plated) determines whetheror not plated based on a standard prepared by comparing brightness of anormal plating area with brightness of a defective plated area.

Referring to the configuration of the machine vision AI system connectedto a sensor, a camera, a light, a control box, a PC or a PLC, it ispossible to develop and continuously learn product defect determinationintelligence on a deep learning-based machine vision platform. Thelearned intelligence is executed on a deep learning-based machine visionplatform to increase the process defect detection rate. Throughcontinuous accumulation of manufacturing data defect detectiontechnology, the machine vision AI system is used as a manufacturingintelligence vision inspection system.

FIG. 6 is a view showing the configuration of a cloud server of a smartfactory.

FIG. 7 is a view showing the configuration of a manufacturingintelligence service system connected to an MES in smart factoryaccording to the present invention.

The manufacturing intelligence service system connected to an MES insmart factory constructs a smart factory in the factory automation (FA)process of an industrial company, together with one or moremanufacturing companies, and through the cloud server 200 connected toat least one Manufacturing Execution System (MES) having a machinevision inspection system that detects defects of products in variousfields, such as wafer, display, and PCB defect inspection, LED chipsemiconductor package inspection, product surface defect inspection, andthe like in a semiconductor production line, the smart factorymanufacturing intelligence service system provides product ID, productinformation and defect information of products of each manufacturingcompany accumulated and stored in the cloud server 200 from the cloudserver 200 to the user terminal 400 through the agent server 300.

The manufacturing intelligence service system connected to an MES insmart factory according to the present invention includes: at least oneManufacturing Execution System (MES) 100 having a machine vision of aproduction line of each manufacturing company to recognize a product IDwhen a barcode, a QR code, or a 13.56 MHz RFID tag is recognized,transmit the product ID through middleware, and provide the product IDand information on the defect (scratch, defect) of the product to acloud server; a cloud server 200 connected to the at least oneManufacturing Execution System (MES) to collect product ID, productinformation and product defect information of the production line ofeach Manufacturing Execution System (MES) and provide product ID,product information and the product defect information to the userterminal 400 through the regional agent server 300; and an agent server300 connected to the cloud server 200, wherein the cloud server 200provides product ID, product information and product defect informationof a machine vision production line of a manufacturing company productto the user terminal 400 through the agent server 300 connected to thecloud server 200.

The user terminal 400 uses a PC, a notebook computer, a tablet PC, or asmartphone.

A barcode, a QR code, or a 13.56 MHz RFID tag is attached to a product,and information on any one among the barcode, the QR code, or the 13.56MHz RFID tag attached to a product recognized by a barcode reader, a QRcode recognizer, or a 13.56 MHz RFID reader is transmitted from theagent server to the user terminal.

The product is attached with any one of a barcode, a QR code, and a13.56 MHz RFID tag.

Additionally, the PC further includes a barcode reader and a recognitionmodule for recognizing a barcode attached to a product when the barcodeis attached to the product.

Additionally, the PC further includes a QR code recognition module forrecognizing a QR code attached to a product when the QR code is attachedto the product.

Additionally, the PC further includes a SW module connected through“product code transmission middleware” from a 13.56 MHz RFID reader whena 13.56 MHz RFID tag is attached to the product.

The machine vision determines defects of a product by analyzing defectssuch as foreign substances, scratches, pattern errors, or the like onthe display of a product surface, and locations of the defects throughimage processing [(1) image acquisition, (2) image binarization, (3)image processing, (4) image analysis, (5) image interpretation] of avision inspection image processing algorithm of a computer connectedthrough the mechanism unit and the camera interface (Frame Grabber,Gigabit Ethernet (GigE), IEEE 1394, camera link, or USB 3.0) connectedto one camera, a line scan camera, or an area scan camera.

The machine vision AI system connected to a sensor, a camera, a light, acontrol box, and a PC or a PLC may develop and continuously learnproduct defect determination intelligence on a deep learning-basedmachine vision platform. Continuously learned intelligence is executedon a deep learning-based machine vision platform to increase the processdefect detection rate.

In the initial stage in which a smart factory of each company is notconstructed at all, a cloud server connected to a ManufacturingExecution System (MES) can be provided information on work management tostore raw materials in a warehouse, management to obtain and place anorder, production plan, production order, work situation, LOTmanagement, process management, quality management that classifiesgood/defective products using machine vision (MV),warehousing/releasing/inventory management, and sales performancemanagement can be provided in the manufacturing industry field.

The Manufacturing Execution System (MES) is used for defect managementof products for recognizing defects and classifying good/defectiveproducts in real-time camera vision inspection monitoring of amanufacturing process.

The encoder measures an exact amount of transfer of a servo motor when aconveyor belt operates in a production line of a factory.

The mechanism unit may further include an inspection target transferrobot for placing an inspection target on the inspection stage (XYstage) by a loader.

The mechanism unit further includes an ID reader for reading a DPM code,a barcode, a QR code, or a 13.56 MHz RFID tag attached to a product asan ID of an inspection target transferred to a conveyor belt of aproduction line in a factory automation process, or an ID of a productplaced on an inspection stage (XY stage) by the loader of the inspectiontarget transfer robot, and transmitting the detected product ID to thecomputer.

In the factory automation (FA) process of an industrial company, defectinformation of products of each manufacturing company, which isaccumulated and stored in a cloud server connected to each MES system toprovide defect information and manufacturing intelligence information ofthe products, is transferred and provided from the agent server to theuser terminal through the cloud server connected to at least oneManufacturing Execution System (MES) having a machine vision inspectionsystem that detects defects of products in various fields, such aswafer, display, and PCB defect inspection, LED chip semiconductorpackage inspection, product surface defect inspection, and the like in asemiconductor production line.

The service platform of the cloud server provides defect determinationmanufacturing intelligence and defect prediction manufacturingintelligence, continuously learns defects of products, and providesvision inspection through product defect image remote learning by usinga deep learning algorithm based on the learning data.

The manufacturing intelligence service system connected to an MES insmart factory is provided with a service platform connected to an edgeplatform including a defect inspection module, a defect determinationmodule, and a learning-purpose manufacturing data transmission modulethrough middleware, in which the defect inspection module reads andtransmits a barcode, a QR code, or a 13.56 MHz RFID tag attached to aproduct with a barcode reader, a QR code recognizer, or a 13.56 MHz RFIDreader to a computer (PC) through middleware so as to be stored, andreads a product ID and provides shape determination inspection/foreignsubstance inspection/scratch inspection on camera image data.

The service platform of the cloud server provides defect determinationmanufacturing intelligence and defect prediction manufacturingintelligence, continuously accumulates and stores defect data, and usesa deep learning algorithm to detect defect data based on the learningdata.

In the cloud server, the deep learning algorithm of machine vision imageanalysis software of each manufacturing company extracts and classifiesfeatures of objects in an image to detect defects, receives and storesdefect information of a product ID in the cloud server, and shares thedefect information, using any one of the algorithms includingCNN(Convolutional Neural Network), R-CNN (Recurrent Convolutional NeuralNetwork), Fast RCNN, Faster RCNN(Region based Convolutional NeuralNetwork), YOLO (You Only Look Once), and SSD (Single Shot Detector).

For reference, the image analysis SW of the camera image data may use i)grayscale image processing, or ii) a deep learning algorithm. Agrayscale image, an RGB image, an HSI image, a YCbCr image, a JPEGimage, a TIFF image, or a GIF image may be applied as the camera image.In an embodiment, a grayscale image is used.

i) Grayscale Image Processing

The image analysis SW converts camera image data (RGB image) intograyscale image data, buffers and stores the grayscale image, providesimage processing and image analysis functions, converts a region ofinterest (ROI) into grayscale, obtains a histogram of an image of theregion of interest (ROI) using a specific threshold of an image of theregion of interest (ROI) [the pixel value of each pixel on the x-axisimage, the number (frequency) of pixel values on the y-axis], binarizesthe image of the region of interest to 0 and 1 on the basis of thethreshold using an Ostu algorithm, performs pre-processing on the imageof the region of interest (ROI) through histogram equalization, obtainsan x-direction derivative and a y-direction derivative using a Sobeledge operator (Sobel mask) or a Canny edge operator, detects an edge(pixels located at the boundary of the object region and the backgroundregion) of the image of the region of interest by convolution ofmultiplying the pixel values of the image by the weight of the Sobelmask and adding them, detects an outline of defective objects in thegenerated edge image by applying a specific threshold, and extractsshape features.

When a threshold method using the Ostu algorithm is used, pixel values f(x,y) are separated into an object region and a background region forthe input image based on a specific threshold. When the pixel valuef(x,y) is greater than the specific threshold, it is determined as apixel belonging to the object region. On the contrary, when the pixelvalue f(x,y) is smaller than the specific threshold, it is determined asa pixel belonging to the background region.

ii) Features of objects in an image are extracted and classified byusing a deep learning algorithm (CNN algorithm, etc.), and defectiveobjects (foreign materials, dents, scratches, etc.) are extracted bycomparing the features of input image with the learning data (foreignsubstances, scratch, etc.) of the defective image accumulated and storedin the learning data DB in accordance with a learning model

The deep learning algorithm of the machine vision image analysissoftware extracts features of the objects in an image or detects adefective image (object detection) using any one of the algorithmsincluding CNN (Convolutional Neural Network), R-CNN(RecurrentConvolutional Neural Network), Fast RCNN, Faster RCNN(Region basedConvolutional Neural Network), YOLO (You Only Look Once), and SSD(Single Shot Detector).

The deep learning algorithm of the machine vision image analysis SW usesa CNN algorithm, and any one of AlexNet, ZFNet, VGGNet, GoogLeNet, andResNet is used. The deep learning algorithm uses the CNN algorithm toextract and classify features of an image (feature extraction), andextracts defective objects (foreign substances, or scratches) bycomparing the features of input image with the learning data of thedefective image accumulated and stored in the learning data DB inaccordance with the learning model.

A multilayer neural network (MLP) having a multilayer perceptron iscomposed of an input layer for inputting a camera input image, n hiddenlayers (Layer 1, Layer 2, Layer 3, . . . ), and an output layer, anddetects defective objects (foreign substances, defects, or scratches,etc.) by extracting image features and classifying objects in an image.

The convolutional neural network (CNN) uses three layers including aconvolutional layer, a pooling layer, and a fully connected layer (FClayer).

A Deep CNN algorithm reduces the amount of image data by repeatingconvolution and subsampling by a convolutional layer and a pooling layerrespectively while moving a mask (e.g., a 3×3 window, filter) having aweight, extracts features robust to image distortion, extracts a featuremap by convolution, and classifies defective objects (foreignsubstances, or scratches, etc.) detected by the learning model of theneural network.

In the image processing using the CNN algorithm, convolutionaccomplishes image processing of input image by using a mask having aweight (e.g., 3×3 window, filter), and a sum obtained after putting amask (e.g., 3×3 window, filter) having a weight in the current inputimage and multiplying the pixel value of the input image by the weightof the mask while moving the mask having a weight in an input image inaccordance with a stride is determined as the pixel value of the outputimage.

Subsampling is a process of reducing the screen size, and max pooling isperformed to select the maximum value of a corresponding area.

The FC layer (Fully Connected Layer) connects to the input terminal ofthe neural network to classify objects by learning.

Currently, the FC layer is configured of a convolution layer of 5 layersand a fully_connected layer of 3 layers.

The size of the image is reduced as the output of the convolutionallayer goes through subsampling by the Max-Pooling Layer, and the outputof the Max-Pooling is classified into classes of the objects in the FClayer (Fully Connected Layer).

As a result, in order to extract defective objects in a camera image, afeature map including object location area and type information isextracted by several convolutional layers in the middle of the CNNstructure, and the size of the feature map decreases while passingthrough the pooling layer, and objects are detected by extracting objectlocation area information from feature maps of different sizes, anddefective objects (foreign substance, or scratch) are classified bycomparing the objects with previously learned data of the learningmodel.

Feature vector x of an image is extracted from input image I of thecamera by using MLP of a multi-layer structure of input layer/hiddenlayer/output layer or a neural network, and output vector h(x) iscalculated from the extracted feature vector x of the image byrepeatedly applying function h_(i)(h_(i-1))=max(O, W_(i)h_(i-1)+b_(i)).

Here, h_(i) is the i-th hidden feature vector, h_(i-1) is the i-1-thhidden feature vector, W_(i) is a weight parameter (a constant value) ofthe neural network circuit, and b_(i) is the bias value of the neuralnetwork circuit.

The input feature vector is set to h₀=x, and when a total of L hiddenlayers exists, h₁, h₂, . . . h_(L) are calculated in order, and thefinal output vector is determined as h(x)=h_(L). In addition, h₁, h₂, .. . h_(L-1) are quantities that are not revealed as an output of thesystem, and are referred to as hidden feature vectors. _(L-1) is theL-1-th hidden feature vector.

The basic structure of the R-CNN extracts Region Proposals, in whichobjects are presumed to exist, from an input image using a RegionProposal generation algorithm called as Selective Search. Each RegionProposal is formed as an image in a bounding box of a rectangular shape,and object classification is performed through the CNN after making thesize of all Region Proposals the same.

The R-CNN slows down the processing speed because one CNN (convolutionalneural network) should be executed for every region proposal, and a lotof time is required for machine learning since a model for image featureextraction, a model for classification, and a model for fixing thebounding box should be learned at the same time.

To solve the processing speed problem of the R-CNN, a Fast R-CNN modelhas been developed. The Fast R-CNN model does not extract features froman input image, but extracts features using RoI Pooling in a feature mapthat has gone through the CNN.

In the Faster R-CNN, a network that combines a method itself ofgenerating Region Proposals inside the CNN as a network structure iscalled as Region Proposal Network (RPN). Through the RPN, the layerperforming RoI Pooling and the layer extracting the Bounding Box mayshare the same feature map.

The Fast RCNN receives an entire image and objects, and acquires afeature map of the CNN for the entire image. The ROI (Region ofInterest) pooling layer extracts a feature vector of a fixed length fromthe feature map for each entity. Each feature vector becomes onesequence through the Fully Connected (FC) layer, and outputs probabilityestimation through Softmax and the position of the bounding box.

Pooling is a sub-sampling process that may lower the resolution of animage by aggregating the statistics of features at various locations,and improves robustness to image deformation such as rotation, noise anddistortion. Two methods of pooling are used maximum pooling and averagepooling.

The convolution layer and the pooling layer are repeated in one CNNclassifier, and layers of various functions may be added according tothe structure. Objects (e.g., foreign substances, scratches, surfacedefects, etc.) may be classified by applying various classifiers (e.g.,SVM classifier) in accordance with the learning data of the learningmodel to the features extracted through the convolution and poolingprocess performed on the input image.

The Faster R-CNN extracts features by passing the whole input imagethrough the convolution layer several times, and the RPN and the RoIPooling Layer share the extracted output feature map. The RPN extractsRegion Proposals from the feature map, and the RoI Pooling Layerperforms RoI pooling on the Region Proposals extracted by the RPN.

A YOLO(You Only Look Once) model may be used for real-time objectrecognition of camera image data by using deep learning.

YOLO divides each image into S×S grids (bounding box), calculatesreliability of each grid, and classifies the class in a way of viewingthe entire image at once by reflecting accuracy when objects in the gridare recognized, and YOLO has performance two times higher than those ofother models owing to the simple process. An object class score iscalculated to determine whether an object is included in a grid. As aresult, a total of S×S×N objects are predicted.

The SSD (Single Shot Detector) model, which is similar to YOLO but showsbetter performance, has a unique advantage in the balance between thespeed and accuracy of detecting objects in an image, and the SSD maydetect objects of various scales as it may calculate a feature map byexecuting CNN on the input image only once.

The SSD is an AI-based object detection algorithm balanced between thespeed and accuracy of detecting objects, in which grids for detectingobjects in a camera image are displayed. The SSD calculates a featuremap by executing a Convolutional Neural Network (CNN) on the input imageonly once. The SSD is performed CNN to extract the feature map with a3×3 filter size to predict probability of grids and objectclassification. The SSD predicts grids after performing CNN. This methodmay detect objects of various scales.

Manufacturing companies adopt an intelligent machine vision solution forfactory automation (FA) process as an edge system, that is the smartfactory manufacturing intelligence service system connected to an MESperforming learning and executing to detect a defect determination andprediction model by using cloud computing in the ManufacturingIntelligence Marketplace (MiraeCIT), thereby providing manufacturingintelligence data from cloud server to user terminals through the agentserver.

Embodiments according to the present invention may be implemented in theform of program instructions that can be executed by various computermeans and recorded in a computer-readable recording medium. Thecomputer-readable recording medium may store program instructions, datafiles, and data structures individually or in combination. Thecomputer-readable recording medium may include hardware devicesconfigured to store and execute program instructions in magnetic mediasuch as storage, hard disks, floppy disks, and magnetic tapes, opticalmedia such as CD-ROMs and DVDs, magneto-optical media such as flopticaldisks, and storage media such as ROM, RAM, flash memory, and the like.Examples of program instructions may include high-level language codesthat can be generated by a compiler and executed by a computer using aninterpreter, as well as machine language codes. The hardware devices maybe configured to operate by one or more software modules to perform theoperations of the present invention.

As described above, the method of the present invention may beimplemented as a program and stored in a recording medium (CD-ROM, RAM,ROM, memory card, hard disk, magneto-optical disk, storage device, etc.)in a form that can be read using computer software.

Although the present invention has been described with reference to aspecific embodiment of the present invention, the present invention isnot limited to the same configuration and operation as the specificembodiment to illustrate the technical spirit as described above, andwithin the limit that does not depart from the technical spirit andscope of the present invention, it can be implemented with variousmodifications, and the scope of the present invention should bedetermined by the claims described below.

What is claimed is:
 1. A manufacturing intelligence service systemconnected to a Manufacturing Execution System (MES) in smart factory,the system comprising: at least one Manufacturing Execution System (MES)having a machine vision of a production line of each manufacturingcompany, recognizing a product ID, for providing the product ID and aproduct information and a defect information including scratch or defectof a product through middleware; a cloud server connected to the atleast one Manufacturing Execution System (MES); and an agent serverconnected to the cloud server, wherein the cloud server provides theproduct ID and the product information and product defect information ofa connected machine vision production line of a manufacturing companyproduct to a user terminal through the agent server.
 2. The system ofclaim 1, wherein the user terminal uses a PC, a notebook computer, atablet PC, or a smartphone.
 3. The system of claim 1, wherein theproduct is attached with any one among a barcode, a QR code, and a 13.56MHz RFID tag.
 4. The system of claim 3, wherein the PC further includesa barcode reader and a recognition module for recognizing a barcodeattached to a product when the barcode is attached to the product. 5.The system of claim 3, wherein the PC further includes a QR coderecognition module for recognizing a QR code attached to a product whenthe QR code is attached to the product.
 6. The system of claim 3,wherein the PC further includes a SW module connected to a 13.56 MHzRFID reader through “product code transmission middleware” when a 13.56MHz RFID tag is attached to the product.
 7. The system of claim 1,wherein the cloud server collects product defect information of aproduction line of a Manufacturing Execution System (MES) of eachmanufacturing company and provides the product defect information to theuser terminal through a provided regional agent server.
 8. The system ofclaim 1, wherein in the cloud server, a deep learning algorithm ofmachine vision image analysis software of each manufacturing companyextracts and classifies features of objects in an image to detectdefects, receives and stores defect information of a product ID in thecloud server, using any one of algorithms including CNN(ConvolutionalNeural Network), R-CNN(Recurrent Convolutional Neural Network), FastRCNN, Faster RCNN(Region based Convolutional Neural Network), YOLO(YouOnly Look Once), and SSD (Single Shot Detector).
 9. The system of claim1, wherein the middleware includes: product code transmission middlewarefor transmitting information on any one among a barcode, a QR code, or a13.56 MHz RFID tag recognized by a barcode reader, a QR code recognizer,or a 13.56 MHz RFID reader to the cloud server; and deep learningmiddleware provided with an atypical defect determination learning modelfor receiving atypical defect process data transmitted from a machinevision system and detecting atypical defective images by comparing theatypical defect process data with the defective image learning dataincluding foreign substances, or scratches accumulated and stored by adeep learning model training system, and transmitting result data offoreign material existence inspection, shape inspection, andnormal/defective determination performed on camera image data by a deeplearning shape determination system and an AI deep learning module tocloud server.